Cat Expression Classifier Using Convolutional Neural Networks¶

This project aims to build a cat expression classifier with convolutional neural networks (CNNs) using PyTorch. This project serves as an introduction to image classification and also dives into the nuances of handling a specific, custom dataset and adapting pre-trained models for our purposes.

Objective¶

The primary objective of this project is to develop a model capable of classifying images of cat faces into one of four moods: alarmed, angry, calm, and pleased. By the end of this tutorial, you will learn how to preprocess image data, leverage transfer learning for image classification, and evaluate a model's performance.

Tools and Techniques¶

We will employ PyTorch, a powerful and versatile deep learning library, to construct our CNN. The model of choice for this tutorial is ResNet18, a robust architecture that is commonly used in image recognition tasks. Given the straightforward nature of our classificaiton problem, ResNet18 provides an ecellent balance between complexity and performance.

Why Transfer Learning?¶

In this tutorial, we utilize transfer learning to take advantage of a pre-trained ResNet18 model. This approach allows us to use a model that has already learned a significant amount of relevant features from a vast and diverse dataset (ImageNet). By fine-tuning this model to our specific task, we can achieve high accuracy with relatively little data and reduce the computational cost typycally associated with training a deep neural network from scratch.

Dataset¶

The dataset comprises images of cat faces, labeled according to their expressed mood. These images are organized into training, validation, and testing sets, each with a corresponding CSV file which maps filenames to mood labels. This guide will walk you through the process of loading, preprocessing, and augmenting this data to suit the needs of our CNN.

The dataset was obtained here.

Let's get started!

Dataset Exploration¶

Listing the Number of Images in Each Set and Visualizing The Set¶

Below we will mount the drive to retrieve the data set files. Then, will use Python's os module to list the number of images in the dataset. This will give us an idea of the size of the set.

Additionally, we will include a flag to tell the model whether we want to train it or to load a previously saved model... this will become clear later.

Finally, we will set up a function to visualize some sample images from each set.

Finally, we will make a dictionary that maps the classes to index-based labels, from the CSV file. We will need this way later, but we will define the dictionary this early on.

Visualizing Some of the Data¶

Let's visualize a few images from each folder to ensure to have a better feel of the data.

We will do this by making a dataframe out of the annotations file where the labels are stored. We will use the test annotations, since this is the smallest dataset, and the other two have the same labels anyway.

Data Preprocessing¶

This steps involves preparing the dataset for training a PyTorch model by resizing, normalizing, and applying data augmentation.

NOTE: At this point, it is important to know what is the model or CNN architecture we will be using. Important aspects to consider include the image size, any data transformations for training and validation, data augmentation techniques, and setting up data loaders later.

In this example, we will use ResNet18. The inputs must follow a specific format, as per the PyTorch ResNet documentation found here:

All pre-trained models expect input images normalized in the same way, i.e. mini-batches of 3-channel RGB images of shape (3 x H x W), where H and W are expected to be at least 224. The images have to be loaded in to a range of [0, 1] and then normalized using mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225].

Tools for Data Preprocessing in PyTorch¶

• torchvision.transforms: Provides common image transformations like resizing, normalization, and augmentation.
• torch.utils.data.Dataset: A base class for creating custom datasets.
• torch.utils.data.DataLoader: Loads and batches data for training.

The Data¶

The data set we are using consists of three folders: train, val, test. Each of them contain a set of images of cats. The labels in this case, are in the form of a CSV file that maps the filename with a one-hot encoding to label the classification of the image, i.e. the cat's mood - alarmed, angry, calm, pleased.

Because this dataset structure is not exactly suitable for the ImageFolder module in PyTorch, whereby labelling is made easier and based on the folder structure, we need to create a custom dataset and loader. Let's get started!

Define Image Transformations¶

• Specify resizing dimensions, normalization parameters, and augmentation techniques (like random rotation, flips, etc.).
• Create separate transformations for training and validation datasets.

With these transformations, the data pipeline will align with common practices for pre-trained models like ResNet18.

Create Custom Datasets and Cada Loaders¶

Given the structure of our dataset, where labels are provided in a CSV file rather than through directory structure, we need to use a custom dataset class. This will allow us to link echc image with its respective label based on our CSV file's structure.

Creating Custom Dataset¶

We will extend the torch.utils.data.Dataset class to create our custom dataset. this class will override the necessary methods to handle our specific dataset setup:

1. Initialization: Load the CSV file and set up the path to the images
2. Length: Return the total number of images
3. Get item: Load each image by index, apply the specified transformations, and parse the label from the CSV data

Now that we have defined the data classes, we can create objects for each of our datasets, as a CustomImageDataset class.

Creating Data Loaders¶

Data loaders in PyTorch provide the necessary functionality to batch, shuffle, and feed the data to your model during training in an efficient manner. They also handle parallel processing using multiple worker threads, which can significantly speed up data loading.

In short, data loaders take the dataset objects and handle the process of creating batches, shuffling the data, and parallelizing the data loading process.

Below we will create a data loaders for our datasets.

In the loader above, we have the following main parts:

• Batch size: typycally set based on the system's memory capacity and how large the model is. A larger batch size can speed up training but requires more memory.
• Shuffle: shuffling helps ensure that each batch sees a varierty of data across epochs, which can improve model generalization.
• Number of workers: this controls how many subproceses to use for data loading. More workers can lead to faster data preprocessing and reduced time to train each epoch but also increases memory usage.

Integration with the Training Loop¶

With the data loaders set up, we are now ready to integrate them into the model's training and validation loops.

The code snippet below shows how this would be done. We will implement the actual integration when we get to the training section.

for epoch in range(num_epochs):
    for images, labels in train_loader:
        # Forward pass, backward pass, and optimize
        outputs = model(images)
        loss = criterion(outputs, labels)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    # Validation step at the end of each epoch
    model.eval()
    with torch.no_grad():
        for images, labels in val_loader:
            predictions = model(images)
            # Calculate validation accuracy, loss, etc.

Model Training¶

Now that the data is ready and properly formatted for input into a neural network, the next step involves setting up and training the ResNet18 model. We will configure the model, define the loss function and optimizer, and implement the training and validation loops.

Next Steps¶

1. Model setup:

• Load the pre-trained ResNet18 model and modify it for our specific classification task (number of classes based on cat facial expressions)

2. Loss function and optimizer:

• Define a loss function suitable for classification, e.g. CrossEntropyLoss
• Set up an optimizer (like Adam or SGD) to adjust the model weights during training based on the computed gradients

3. Training loop:

• Implement the loop that processes the data through the model, computes the loss, updates the model parameters, and evaluates the model performance on the validation dataset periodically

4. Monitoring and saving the model:

• Track performance metrics such as loss and accuracy
• Implement functionality to save the trained model for later use or further evaluation

Model Setup¶

In this section, we'll configure a ResNet18 model to suit our specific classification task. Since the model is originally designed for ImageNet with 1000 classes, we'll adapt it for our use case, which involves classifying images into four mood categories (alarmed, angry, calm, pleased).

Import the Necessary Libraries¶

Load and Modify the Pre-trained ResNet18¶

We will load a pre-trained ResNet18 model and modify its final layer to suit our classification needs. This is known as transfer learning, and it is a technique that uses a pre-trained model and leverages its learned parameters to focus on a similar, more specific task. This is a powerful technique, since it uses the existing knowledge (such as edges and features) so that the new classification task is more robust, and faster to tune to the specific task.

Understanding Transfer Learning¶

Transfer Learning is a powerful technique in machine learning where a model developed for a particular task is reused as the starting point for a model on a second task. It's especially popular in deep learning given the vast compute and time resources required to develop neural network models on large datasets and from scratch.

Why Use Transfer Learning?¶

1. Efficiency: transfer learning allows us to leverage pre-trained networks that have already learned a good amount of features on large datasets. This is beneficial as it can drastically reduce the time and computational cost to achieve high performance.
2. Performance: models trained on large-scalr datasets like ImageNet havve proven to generalize well to other datasets. Starting with these can provide a significand head-start in terms of performance.

Applying Transfer Learning¶

• Model adaptation: for our specific task fo classifying cat moods, we take a pre-trained ResNet18 model and tailor it to our needs. The pre-trained model brings the advantage of learned features from ImageNet, a vast and diverse dataset.
• Feature extraction: by freezing (i.e. keeping the weight values as they are) the pre-trained layers, we utilize them as a feature extractor. Only the final layers are trained to adapt those features to our specific classification task.

Model Setup with a Custom Classifier¶

We have mentioned replacing the funal layer(s) as a transfer learning techniques. In this case, we replace the final fully connected (fc) layer of ResNet18 with a different layer which will suit our need to have 4 classes. Additionally, we will replace this fc layer with a more complex classifier portion, which involves adding additional layers such as ReLU for non-linearity, and dropout for regularization to prevent overfitting.

• ReLU Activation: introduces non-linearity into the model, allowing it to learn more complex patterns.
• Dropout: Randomly zeros some of the elements of the input (to the layer, not input to the model) tensor with probability $p$ during training, which helps prevent overfitting.

Let's implement this classifier in our transfer learning setup.

Loss Function and Optimizer¶

For our classification task, we need a loss function that effectively measures the discrepancy between the predicted labels and the actual labels. Since we've configured out model outputs to be class indices (from our dataset's one-hot encoded labels), we'll use CrossEntropyLoss, which is ideal for such clasification tasks.

We'll par this with the Adam optimizer, which is known for its efficiency in handling sparse gradients and adaptive learning rate capabilities, making it well-suited for this task.

Let's set up our loss function and optimizer.

Training and Validation Loops¶

Now, let's write the code to train and validate our model. This involves running the model over several epochs, making predictions, calculating loss, updating the model parameters, and evaluating the model's perfomance on the validation dataset.

• Training loop: here, the model learns by adjusting its weights based on the calculated loss from the training data
• Validation loop: validation occurs post the training phase in each epoch and helps in evaluating the model's performance on unseen data, ensuring it generalizes well and doesn't overfit

Saving/Loading the Model¶

Save the Trained Model¶

If we have performed training, we can save the model to use next time, so that we can avoid re-training everytime we run the notebook.

We will implement this part as an if statement, that would run the training loop and save the model if we choose to, otherwise, we would just load the model weights.

Model Evaluation¶

Now that the model is trained, the next step is to evaluate its performance more thoroughly, and possibly improve it based on the insights gained.

Evaluating the model involves checking the accuracy and also looking at other metrics like precision, recall, and F1-score, especially if the dataset is imbalanced or if specific classes are more important than others.

Model Evaluation on Validation Set¶

After training a machine learning model, it's crucial to evaluate its performance comprehensively. Here, we will detail three key diagnostic tools"

1. Confusion matrix
2. Plotting training and validation losses
3. Visualization of the predictions

Confusion Matrix¶

A confusion matrix provides a detailed breakdown of the model's predictions, showing exactly how many samples from each class were correctly or incorrectly predicted as each other class. This is crucial for understanding the model's performance across different categories.

Plotting Training and Validation Losses¶

Plotting the training and validation losses over epochs allows us to monitor the learning process, identifying issues such as overfitting or underfitting.

Visualization of the Predictions¶

Visualizing model predictions on actual data points provides immediate qualitative feedback about model behavior. It helps identify paterns in which the model performs well or poorly, revealing potential biases, underfitting, or overfitting.

Evaluation of Model Performance¶

Our Cat Expression Classifier, built on a modified ResNet18 architecture, demonstrates a promising ability to classify cat expressions into four categories: alarmed, angry, calm, and pleased. Here, we provide a detailed analysis of the model's performance based on our the training and validation efforts.

Overall Performance Metrics¶

The model achieves an overall accuracy of 68.18% on the validation set. This is a decent foundation but indicates room for further refinement, especially in distinguishing between expressions that share subtle features. Here is a breakdown of the key performance metrics:

• Precision: Measures the accuracy of positive predictions. for example, the pleased category shows high precision, indicating that the model reliably identifies this expression.
• Recall: Reflects the model's ability to identify all relevant instances of a class. The angry category has a high recall, suggesting that the model effectively captures most of the angry expressions.
• F1-Score: Balances precision and recall and is particularly useful in scenarios where class distribution is uneven.

Confusion Matrix Insights¶

the confusion matrix provides a granular view of the model's performance across the different classes. It highlights specific areas where the model performs well and others where it struggles, such as:

• Misclassifications between alarmed and angry suggest that the model may be conflating these expressions due to their similar features.
• The high accuracy in identifying pleased expressions shows that distinct features of this mood are well captured by the model.

Training and Validation Losses¶

The training and validation loss plots reveal the learning dynamics over the epochs:

• A steady decrease in training loss indicates that the model is effectively learning from the training data.
• The pattern of validation loss provides insights into the model's generalization ability. Increases in validation loss suggest moments where the model might be overfitting to the training data.

Trying out VisionTransformers¶

Data Transformations for Vision Transformer¶

When transitioning from a CNN like ResNet18 to a Vision Transformer (ViT), it's essential to evaluate whether the existing preprocessing steps - particulary the data transformations - are suitable for the new model architecture. For ViT , wed must consider their unique handling of image data, which relies on dividing the image into fixed-size patches and understanding global dependencies through self-attention mechanisms.

For this exercise, we will maintain the same transformations we have previously defined.

The decision to retain the initial transformations is based on the principle of consistency and the minimal impact expected by changing model architectures regarding how images are scaled and augmented. The chosen transformations ensure that the images are adequately prepared for the neural network without introducing complexities or distortions that could hinder the learning of global patterns, which are vital for Vision Transformers due to their reliance on self-attention mechanisms.

Additionally, maintaining these transformations allows for a more straightforward comparison between the ResNet18 model and the Vision Transformer model, as any changes in model performance can more confidently be attributed to the architectural differences rather than changes in data preprocessing.

Load Pre-Trained Vision Transformer Model¶

First, we need to load the ViT model that has been pre-trained on a large dataset. We'll then adapt the classifier head to our needs, which is classifying cat moods into four categories.

Freezing the Encoder Layers¶

Freezing the encoder layers prevents their weights from being updated during training, which means they retain the knowledge they have already gained from ImageNet. We only want to train the classifier that we will modify to our specific task.

Modify the Classifier¶

The standard ViT model includes a classifier at the end (usually named heads in torchvision models), which is a linear layer designed for the original classification task, e.g. 1000 classes for ImageNet. We will replace this with a new classifier suited for our task (4 cat moods).

Define Loss Function and Optimizer¶

Now, define the loss function and an optimizer. Since we are only training the classifier layer, ensure the optimizer is set to only update the parameters of the classifier.

Training Loop¶

Here, we revisit the training process, adapting our previously established procedures to the Vision Transformer (ViT) model. Much like our approach with Resnet18, we utilize a similar training loop structure to ensure consistency and comparability. the core steps of training - forward pass, loss computation, backward pass, and parameters update - are maintained, but they are now applied to a differently structured model that leverages self-attention mechanisms rather than convolutional layers. This section briefly outlines these steps, focusing on any adjustments specific to the ViT to optimize it for our cat mood classification task.

Model Evaluation¶

Confusion Matrix¶

The confusion matrix below shows the following results:

• Class alarmed: moderate confusion with other classes, indicating difficulty in distinguishing alarmed from other moods.
• Class angry: high accuracy, showing that `angry is well-recognized, with few misclassifications
• Class calm: some confusion, particularly with pleased, suggesting similar features or expressions between these moods that the model confuses
• Class pleased: best performance, indicating clear distinguishing features that the model learnes effectively

Considerations on Data Quality¶

Throughout the development and evaluation of our models, it has become evident that the quality of the dataset significantly impacts the classification accuracy. Certain misclassifications observed, such as the confusion between 'pleased' and 'calm' or 'alarmed' and 'angry,' suggest that the labels may not always align perfectly with the visual cues present in the images. This discrepancy can stem from subjective interpretations of cat expressions during labeling. Improving the dataset by refining the labeling process, possibly with the assistance of animal behavior experts, or by curating a more consistently labeled dataset could enhance model performance. Enhancing data quality would help in training more accurate and reliable models, thereby increasing the robustness of the classification outcomes.

Classification Report¶

From the classification report, we can draw the following conclusions:

• Class pleased shows the highest precision, indicating a high rate of true positive predictions
• Class angry has the highes recall, suggesting effective identification of this mood
• Classes angry and pleased show high F1-scores, indicating robust performance.

Overall Accuracy¶

The model achieves and accuracy of 74.55%, which is a solid performance but suggests room for improvement, particularly in reducing misclassifications among less distinct moods.

Training and Validation Losses¶

The plot of training loss shows a consistent decrease, indicating that the model is effectively learning from the data. The vlaidation loss decreases alongside the training loss but begins to plateau, suggesting that the model might be nearing its learning capacity with the current configuration and dataset.

Conclusion¶

This tutorial guides you through creating a cat expression classifier using convolutional neural networks with ResNet18 and later with a Vision Transformer (ViT). It demonstrates how to apply transfer learning to improve efficiency and accuracy with limited data.

The guide is structured to provide clear steps and practical examples for each phase of the project, from data preprocessing and model training to evaluation. By breaking down complex concepts and processes into manageable parts, it ensures that readers can easily follow along and apply these techniques to their projects.